1. The Field of the Invention
The present invention relates to computer programming, and more particularly to regression testing of computer software from the development through maintenance phases.
2. Present State of the Art
Complex, mission-critical business applications typically contain large numbers of on-line transactions, each of which consists of considerable functionality. Software application quality largely depends on extensive testing of the functionality of each of the on-line transactions to ensure completeness and correctness. Software developers recognize that testing of application software is an ongoing or continuous process that can be integrated into every phase of the application software life cycle including the design phase, development phase, unit testing phase, systems testing and integration phase, quality assurance phase, customer installation and support phases as well as the application maintenance phase.
Present industry trends dictate that approximately eight hours are required to thoroughly test an application transaction if the individual performing the test both understands the transaction and has data available for testing. Therefore, if an application contains 500 transactions, such an application would require approximately 4,000 hours to comprehensively test the applications transactions, if there are no errors present. Furthermore, in order to properly test an application, when an error is discovered and fixed, the entire test must be reperformed precisely in the same manner using the same data to ensure that the repair for the discovered error did not inadvertently introduce another error into the application.
Additional shortcomings are also present in the traditional manners in which software applications have been tested. For example, due to the large number of hours required for properly testing the transactions of an application, dedicated testing staff have generally been required. It is both impractical and expensive to dedicate large numbers of staff to perform such manual testing processes. Furthermore, in order to properly test software application transactions, it is important to capture product content knowledge, usually best appreciated by the transaction designer, for use in selecting test data. To do so ensures a more robust testing of application transactions. However, to transfer such knowledge and skill from the originating entity, such as the transaction designer, to a transaction tester requires additional time and resources for such a knowledge transfer. Additionally, manual testing of application transactions by humans may inject additional errors such as the disregarding of a test sequence or the inadvertent overlooking of test results. Furthermore, one of the more overriding issues in software transaction testing occurs when changes are made to transactions thereby obsoleting a substantial portion of the test information.
Traditional automated application testing strategies have only been partially successful in minimizing the manual nature of transaction testing. For example, one type of automated application testing employs a keystroke-capture technique (FIG. 1) wherein the computer memorizes the keystroke and mouse movements performed by the test operator enabling the computer to replay the test at a later time for retesting the application. Such keystroke-capture techniques fall short for many reasons. For example, keystroke-capture techniques rely on the graphical user interface design thereby precluding testing of transactions prior to the implementation of the graphical user interface. That is to say, for testing strategies employing keystroke-capture techniques any changes, even subtle changes, to the graphical nature of the transactions renders the prior tests unusable. Furthermore, the test data generated by a keystroke-capture technique is normally unmaintainable and incomprehensible, and therefore, any changes to the test data require the re-creation of the test to enable the entire test to be re-recorded for subsequent playback.
Additionally, keystroke-capture testing techniques typically do not generate output but rather sequentially execute the transaction undergoing test. Therefore, to evaluate the behavior of the test, a test operator must physically observe the test being executed to verify the results. Furthermore, such keystroke-capture techniques yet introduce the human error aspect into transaction testing as the test operator is required to observe and visually verify the integrity of the test. Finally, keystroke-capture testing techniques are difficult and even impossible to document and verify as hard copies of test data are unavailable, or are at least incomprehensible as they are a series of cryptic graphical commands addressed to the graphical user interface of the application undergoing testing.
Preparation of test data is yet another shortcoming of traditional automated testing techniques. Presently, there does not exist an acceptable way to develop test data for traditional testing methods except by using traditional manual techniques for developing such tests. Furthermore, once test data is developed, maintaining such test data is virtually impossible as it has a manually developed origin. Therefore, application programmers have heretofore invented or created non-meaningful data that often does not actually properly exercise the transaction or application undergoing the testing process. While present testing strategies are lacking in proper discipline, the tools and strategies necessary for improving testing methods and strategies have heretofore been nonexistent.
While present industry trends such as the heterogeneity of computer architectures and communication standards exacerbate the testing discipline, other modem trends such as the low-cost and increased speed of computer resources facilitate the economical implementation of an advanced application testing philosophy. Thus, what is needed is a method for enabling robust test data to be created such that changes to the transaction whether during the design, maintenance or other phase do not render the test data unusable or obsolete. What is yet needed is a method for representing the test results in a report that is both comprehensible and comprehendible that enables efficient regression testing of a transaction throughout its life cycle.